DOCUMENT RESUME 

ED 362 976 EA 025 343 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



Ziobrowski, Eileen Merriam; Newman, Dianna L, 
Teacher Perceptions of Program Evaluation: 
Opportunities in Shared Decision Making* 
Apr 93 

lOp* ; Paper presented at the Annual Meeting of the 
American Educational Research Association (Atlanta, 
GA, April 12-16, 1993), 

Speeches/Conference Papers (150) — Reports - 
Research/Technical (143) 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



MFOl/PCOl Plus Postage, 

Decision Making; Elementary Secondary Education; 
'^Participative Decision Making; '^Program Evaluation; 
School Involvement; ^''Teacher Influence; ''Teacher 
Participation; '^Teacher Role 
*New York 



ABSTRACT 

This paper presents findings of a study that examined 
teachers* perceptions of opportunities for and their willingness to 
participate in decision making regarding program evaluation* 
Questionnaires were mailed to a random sample of 477 K-12 teachers in 
the New York public school system, eliciting an unspecified number of 
responses* Overall, teachers indicated that they had limited 
involvement in program evaluation, but would like to be more 
involved, especially in decisions related to evaluation design and 
use* Teachers perceived the most opportunity for involvement in 
program evaluation to be in developing plans for implementation; they 
identified the lowest area of opportunity as deciding what 
information should be directed to various audiences* Decisions that 
involved making recommendations constituted the most preferred area 
of involvement; the least preferred was identification of constraints 
on the evaluation* An implication is that evaluators should recognize 
that teachers feel disenfranchised from the evaluation process and 
that efforts to include teachers in decision making should be 
developed and encouraged* Two tables are included* (LMl) 
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Background of the Study 

Involving teachers as stakeholders in program evaluation has long 
been thought to enhance evaluation's capacity to generate information- 
based change in school planning and policy development (Archbald & 
Newmann, 1988; Tuckman, 1985; King & Pechman, 1983; Wolf, 1979), 
Research, however, indicates that teachers have negative views of 
evaluation and see it as a "necessary evil" (David, 1981; Stuff lebeam 
and Welch, 1986) • This has led to the conclusion that teachers' 
viewpoints affect and frequently result in under-utilization of the 
evaluation process and evaluation results (King & Thompson, 1983; 
Williams & Banks, 1981) . As a consequence, models of evaluation in 
education have begun to include the need to involve teachers in 
program evaluation processes in their schools (Alkin, 1990; Cuba & 
Lincoln, 1989) . 

Stakeholder based evaluation calls for involving individuals who 
"have something at stake in the evaluand in determining how an 
evaluation is to be focused, what information is to be collected, and 
how interpretations of the evaluation are to be made" (Lincoln & 
Cuba, 1990) • This involvement can be equated with a shared decision 
making process (Greene, 1988) • Both shared decision maKing and 
stakeholder based evaluation models promote the involvement of 
individuals in decisions in that will directly affect them. Both 



propose that this involvement will yield: 1) better decision, 2) 
greater commitment to decisions, and 3) political/social empowerment 
of the individuals (Greene, 1988; Whitmore, 1988; Lincoln St Guba, 
1990) . Theorists and practitioners agree on what "should be 
happening"; however, there is very little research on what "really is 
happening". While research does indicate that teacher involvement in 
decision making in evaluation is an important factor in ensuring 
desired outcomes, very little is known about teachers' true 
participation in evaluation decision making or their desire to be 
involved in decision making. 

At this time, many states, including New York, are mandating that 
schools implement shared decision making procedures; one suggested 
area of implementation is program evaluation. No specific directions 
or procedures exist, however, on how this implementation is to take 
place. No information is available to guide administrators, program 
planners and evaluators on how, or if, shared decision making will 
impact program evaluation. 
Purpose of the Study 

The purpose of this study was to investigate teacher perceptions 
of opportunities for and willingness to participate in decision making 
in program evaluation. Perceptions of opportunity and willingness 
were examined in two ways: 1) the actual level of opportunity to be 
involved and 2) the level at which teachers desire to be involved. 
"Decision making" in program evaluation was operationalized to mean 
the actual decisions which are made related to the 1) design, 2) 
implementation and 3) use of program evaluation. "Involvement in 
decision making" refers to the type of participation. For the 



purposes of this study, assessment of involvement used a continuum 
developed by Conway (1984) which delineated level of participation as 
ranging from "no involvement" through "shared che decision making with 
administrators" . 
Methodology 

Sample > The subjects for this study consisted of 477 teachers, 
randomly selected from the membership list of a major state teachers 
union • All respondents were certified to teach in the state of New 
York and were currently employed in K-12 public school systems. 
Examination of the demographics ( gender, education and certification 
levels) indicated that the respondents were representative of New York 
state and national teaching populations. 

Instrumentation . A mail-out mail-back survey, composed of two 
sections, was used to assess teachers' perceptions of program 
evaluation as it related to shared decision making. The first section 
measured teachers' perceptions of current opportunities to be involved 
in decisions pertaining to program evaluation. For this section, 
Likert-type stems were developed that identified eighteen decision 
situations in which teachers could be involved when program 
evaluations were conducted. Six of the stems represented design 
decisions, Six represented evaluation process decisions and six 
represented evaluat'on recommendation/use decisions. For each 
decision setting, respondents were asked to indicate their perception 
of the degree of opportunity found for current involvement for their 
district using a four point scale ranging from "no involvement" to 
"making the decision jointly with the administrator". The second 
section of the survey utilized the same eighteen items but asked 



respondents to indicate their preferred level of involvement using the 
same four point scale ranging from "no involvement" to "making the 
decisions jointly with administrators". Development and validation of 
the eighteen stems included a survey and follow-up interviews of a 
representative national sample of Directors of Evaluation for school 
districts, educational administrators, teachers involved in the 
implementation of shared decision making, and theorir*ts in evaluation. 
Coefficient alpha reliability for the two scales was greater than .90. 
Results — 
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A series of descriptive and empirical analyses were used to 
examine the data. Repeated measures analysis of variance, using mean 
responses as the dependent variable, indicated significant differences 
between responses on opportunities for involvement and desire to be 
involved in shared decision making in evaluation. Significant 
differences were also found among the thee types of evaluation 
decisions. Examination of the means (Tables 1 and 2) indicated that 
teachers perceived the most opportunity for involvement in evaluation 
decision making to be in developing plans for implementing 
recommendations; tne lowest area of current opportunity was in 
deciding what information should go to various audiences. The most 
preferred area of involvement represented decisions that involved 
making recommendations; the least preferred area of involvement was 
identification of constraints on the evaluation. Overall, teachers 
indicated that they currently had only limited involvement in shared 
decision making related to program evaluation, that th<>y were either 
not involved or only asked to supply information. Teachers did 
indicate, however, that they would like to be more involved in 
decisions related to program evaluation, especially decisions related 
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to design and use. In these areas, teachers indicated that they would 
like to provide information and be part of the decision team. 

Conclusions and Implications 

The findings of this study indicate that the philosophy of shared 
decision making, as implemented in many K-12 educational settings, 
will have an isipact on the design, process and use of program 
evaluation. While many settings have not yet included program 
evaluation decisions as one of the areas for shared decision making, 
the stake-holder based approach to evaluation is a fore runner to this 
inclusion. The results of this study indicate that teachers have 
limited involvement in program evaluation, but would like to have 
more, especially in the areas of design and use* 

The results of this study have implications for school 
evaluators, educational administrators, and teachers working toward 
the implementation of shared decision making, Evaluators need to be 
aware of the fact that, despite the increased use of stake-holder 
models of evaluation, teachers feel disenfranchised from the 
evaluation process. If evaluation results are to be meaningful and to 
be useful to administrators, teachers and students, efforts to involve 
teachers in decision making should be developed and encouraged. 
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Table 1 

Current Opportunity for Involvement in Evaluation Decision^* 

Ranked by Item Means 



Item 



Design Decisions 



Defining measurable outcomes/goals of 
programs to be evaluated 

Determining key questions or issues to address 
in e'^'aluating programs 

Deciding when programs need to be evaluated 

Identifying necessary resources (equipment, 
facilities) for program evaluation to take place 

Determining the best evaluation design for a program 

Deciding who conducts program evaluation 



2.076 

2.017 

1.956 
1.948 

1.890 
1.765 



sd 

1.106 

1.081 

1.045 
1.049 

1.058 
0.975 



Process Decisions 

Deciding ways to measure program implementation and 1.933 1.098 
outcomes (e.g., questionnaires, observation tools, 
surveys, interviews, tests, etc.) 

Developing questions/content to be included in measures 1.926 1.095 

Deciding when, where, and how data will be collected 1.813 1.057 

Deciding how data will be analyzed 1.726 1.030 

Deciding who/what will be sampled 1.683 0.985 

Identifying constraints to program evaluation 1.679 0.914 



Use Decisions 

Developing plana for implementing recommendations 

Developing recommendations after reviewing 
evaluation results 

Determining if prograun objectives have been achieved 

Deciding what groups will get evaluation results 
( staff, parents, board members, community) 

Deciding how to present evaluation results to 
different groups 



2.128 
2.118 

2.114 
1.723 

1.688 



1.120 
1.096 

1.074 
1.005 

1.012 



Deciding what information different groups will get 



1.670 



0.964 
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Table 2 

Preferred opportunity for Involvement in Evaluation Decisions 

Ranked by Item Keana 



Item 

Design Decisions 

Deciding when programs need to be evaluated 

Determining key questions or issues to address 
in evaluating programs 

Defining measurable outcomes/goals of programs 
to be evaluated 

Determining the best evaluation design for a program 

Deciding who conducts program evaluation 

Identifying necessary resources (equipment, 
facilities) for program evaluation to take place 



3.669 
3*650 

3.648 

3.605 
3.584 
3.519 



sd 

0.577 
0.619 

0.62.1 

0.685 
0.656 
0.731 



Process Decisions 

Deciding ways to measure program implementation and 3.593 
outcomes (e.g., questionnaires, observation tools, 
surveys, interviews, tests, etc.) 

Developing questions/content to be included in measures 3.576 

Deciding who/what will be sampled 3.572 
(e.g., which students will be tested, which classrooms 
will be observed, which teachers will be interviewed) 

Deciding when, where, and how data, will be collected 3.479 

Deciding how data will be analyzed 3.451 

Identifying constraints to program evaluation 3.339 
(funding, political/ legal, ethical) 



0.658 

0.681 
0.701 

0.784 
0.807 
0.843 
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Use deciaio^is 

Developing recommendations after reviewing 
evaluation results 

Determining if program objectives have been achieved 

Developing plana for implementing recommendations 

Deciding what groupa will get evaluation results 
( staff, parents , board members, community) 

Deciding what information different groupa will get 

Deciding how to present evaluation results to 
different groupa 
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3.712 

3.681 
3.680 
3, 557 

3 .523 
3.476 



0.579 

0.612 
0.624 
0.715 

0.743 
0.775 



